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Programmable Melody Generator 



Technical Field of the Invention 

This invention generally relates to synthesis of musical sounds. 

Description Of The Related Art 

The synthesis of musical notes and melodies from a stored data 
representation is commonly used in a variety of digital systems, such as: 
instrumental keyboards, toys, games, computers, and wireless communication 
devices. One method of digitally representing musical notes is the Musical 
Instrument Digital Interface (MIDI) and is a standard for communicating between 
keyboards, soundcards, sequencers, effects units, and many other devices, most of 
which are related to audio or video. A synthesizer generates musical tones in 
response to a MIDI file by controlling a bank of tone generators. The tone 
generators may be discrete oscillators or simulated electronically, often by using a 
digital signal processor with spectrum models for tone restitution. Another way of 
making synthetic music is by using samples recorded from actual instruments. 

Many different types of processors are known, of which microprocessors are 
but one example. For example, Digital Signal Processors (DSPs) are widely used, in 
particular for specific applications, such as mobile processing applications. DSPs 
are typically configured to optimize the performance of the applications concerned 
and to achieve this they employ more specialized execution units and instruction 
sets. Particularly in applications such as mobile telecommunications, but not 
exclusively, it is desirable to provide ever increasing DSP performance while 
keeping power consumption as low as possible. 

To further improve performance of a digital system, two or more processors 
can be interconnected. For example, a DSP may be interconnected with a general 
purpose processor in a digital system. The DSP performs numeric intensive signal 
processing algorithms while the general purpose processor manages overall control 
flow. The two processors communicate and transfer data for signal processing via 
shared memory. 
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Particularly in portable equipment such as wireless digital assistant devices, 
minimizing power consumption is important. Accordingly, there is needed a system 
and method for synthesizing quality musical tones that is computationally efficient. 

Summary of the Invention 

In accordance wi th a first embodiment of the invention, a method is provided 
for synthesizing music in a, digital system. An analysis digital waveform is first 
accessed that has duration, a pitch, an attack portion and a decay portion. The 
duration and pitch for a note to be synthesized is determined. A set of timing 
; analysis waveform is determined such that the timing marks 

correspond to periodicity of the analysis digital waveform. A second set of timing 
marks is computed for the synthesis waveform such that the second timing marks 
correspond to periodicity of the synthesis waveform. Samples are calculated for 
each peijod defined by adjacent timing marks using samples selected from a 
corresponding period in the analysis waveform defined by adjacent timing marks to 
: > form the ; synthesized digital waveform. , ^ , 

, In a first embodiment, the samples are calculated by first calculating a set of 

samples for a period m using a first cosinous window, then calculating a set of 
..^samples for a period m r l using a second cosinous window; and then combining the 
set of samples for period m and the set of samples for period m-1 using a weighting 
function. 

In another embodiment, samples are calculated by occasionally reversing a 
selected one of the set of samples before the step of combining the sets of samples. 

In another embodiment, an analysis waveform is used to synthesize a range 
of at least two octaves for an instrument. 

Another embodiment of the invention is a digital system that has a memory 
for holding a plurality of instrumentally correct digital waveforms corresponding to 
a plurality of instruments. There is a first processor connected to the memory and 
the first processor is operable to store a musical score in the memory. There is a 
second processor connected to the memory and the second processor is operable to 
synthesize a melody signal in response to the musical score using the method 
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described above. There is also an audio device connected to the second processor for 
playing the synthesized melody sigrial. 

Brief Description of the Drawings 

Particular embodiments in accordance with the invention will now be 
described, by way of example only, and with reference to the accompanying 
drawings in which like reference signs are used to denote like parts and in which: 

Figure 1 is a representative waveform illustrating an attack portion and a 
decay portion; . , • > v .... 3 - 

Figure 2A is an illustration of ^n analysis waveform envelope of a sample 
music note of Figure 1 that will formlM basis'cif miisib Aotes synthesized according 
to aspects of the present invention;-' ■"' ' ^ 

Figure 2B is an illustration of a synthesis waveform envelope of a 
synthesized music note having a shorter duration than the analysis waveform of 
Figure 2A; vxr- ^."^..i . 

Figure 2C is an illustration of a Synthesis waveform envelope of a 
synthesized music note having a loiigef duration thdn thei analysis waveform of 
Figure 2A; : : : ' ;V - • " ? : • ' " ' 

Figure 3 is an illustration bf a higher pitched note that is synthesized from a 
lower pitched analysis waveform, illustrating replication of eui'Vlanalysis period, 
according to an aspect of the present invention; 

Figure 4 is a an illustration of another embodiment of the invention 
illustrating the use of two windows to form a synthesized waveform; 

Figure 5 is a flow chart illustrating steps for synthesizing a note according to 
aspects of the present invention; 

Figure 6A is a flow diagram illustrating how a set of analysis waveforms are 
collected; 

Figure 6B is an illustration of a set of analysis waveforms for several 
different instruments; 

Figure 7 is a block diagram of a digital system that includes an embodiment 
of the present invention in a megacell core having multiple processor cores; 
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Figure 8 is a flowchart illustrating synthesis, of a melody on the digital 
system of Figure 7 according to an aspect of the present invention; and 

Figure 9 is a representation of a wireless telecommunications device 
incorporating an embodiment of the present invention. 

Corresponding numerals and symbols in the different figures and tables refer 
to corresponding parts unless otherwise indicated. 

Detailed Description of Embodiments of the Invention 

[01] / -Previous solutions for synthesizing music have used large memory 

-sizes or high processing rates to produce good quality, synthesized music. When 
processing rate is optimized, , large over-sampled memory arrays are used to store 
multiple sound samples. When memory size, is optimized, then complex digital 
filtering and interpolation schemes are used which require high processing rates. 
, Li many cases, the synthesized sound is degraded due to digital down-sampling. 
The sound spectrum is shifted, in order to reach a . targeted sound pitch. The 
resulting sound tone is then disturbed because the short-term spectrum is also 
shifted. 

[02] t . ^ : v A< method for synthesizing music has now been discovered that solves 
the .tradeoff bejfcween large memory or high processing loads.. This method 
generates the correct pitch with half-tone precision using prerecorded samples that 
dp not have the same pitch. This operation is done with, only a few arithmetic 
operations per digital sample and using a small data buffer size in order to let the 
music be played on low power portable devices, such as a wireless telephone. The 
novel methods that will now be described make use of a mathematical technique 
similar to one described in a paper entitled "Time-Frequency Representation of 
Digital Signals and Systems Based on Short-Time Fourier Analysis" by Michael 
Portnoff, IEEE Transaction on Acoustics, Speech, and Signal Processing, Vol. ASSP- 
28, No 1 Feb 1980. 

[03] Figure 1 is a representative waveform illustrating an attack portion 

and a decay portion of an instrumentally correct digital waveform 100, according to 
an aspect of the present invention. This waveform is a digitally sampled waveform 
of a single note struck on a musical instrument, such as a piano. The current 
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embodiment of the invention is concerned with wireless telephony, in which the 
bandwidth of the sound reproducing system is generally limited to approximately 
four kilohertz, therefore a sampling rate of eight kilohertz is used. The following 
descriptions all assume this sample rate, however, it should be understood that the 
invention is equally useful for higher quality sound synthesis systems. In such 
systems, the sample waveforms would typically be sampled at higher rates, such as 
twenty kilohertz or higher, for example. 

[04] Digital waveform 100 is a ^iirglfe periodic: hot£: The duration of the 

period is the inverse of the fundaMebial frequency arid is deiiotM as Ta. Waveform 
100 has a fundamental frequency of 500 hertz, therefore its period Tia is 2 ins and 
each period is sampled app^^ 

[05] Time line 104 provides a tiine references for the following description. 

A set of timing marks, represented by '106a, b, are marked on time line 104 and 
correspond to period boundaries of waveform 100. Thus; for waveform 100 each 
period Ta bounded by adjacent timing marks 106a, 106b includes sixteen digital 
samples. For a non-periodic wavefbrmi ^ timing marks can bfe assigned at regular 
intervals. 

[06] A first "portion of digital wavefbrai 100 that ihcliides a set of timing 

marks denoted as Tl is referred to as the attack portion: This corresponds to the 
initial sound produced by a stringed instrument when a string is hit or plucked, or 
by percussion instrument when struck, or by a wind instrument when a note is 
sounded. Typically, the attack portion builds up to crescendo £nd then subsides. A 
second portion of digital waveform 100 that includes a set of timing marks denoted 
as T2 is referred to as the decay portion. During the decay portion, the string 
vibration slowly dies out or is damped, the percussion vibration slowly dies out or 
the wind tapers off. 

[07] The relative duration of the Tl phase and the T2 phase depends on the 

type of instrument. For example, a flute generally produces a strong short attack 
with a relatively long decay, while a piano produces relatively long attack phases 
and shorter decay phases. For lower notes produced by longer strings, the decay is 
longer due to the longer string, resonance, etc. Advantageously, by using 
instrumentally correct recordings from actual instruments, this embodiment of the 
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invention captures nuances of the musical instrument, such as reverberation, 
damping, etc. Therefore, melodies can be synthesized that recreate the tonal 
characteristics of the original instruments. 

[08] Because of the variation in attack phase and decay phase relative 

duration times, each digital waveform is visually inspected by displaying the 
waveform on a display device.: A boundary between Tl and T2 is then selected 
based on the inspection and included with-the digital file. The set of timing marks 
are also included with the digital file associated with digital waveform 100. 
[09] ; Referring still to Figxire 1,^ envelope 102 is representative of waveform 

100 and will be used for clarity in the-fpllpwing- description. 

[10] Figure 2A is an illustration of an analysis waveform envelope 200 of a 

sample music note representative of waveform 100 of figure 1 that will form the 
basis of music notes synthesized according to aspects of the present invention. 
Waveform 200 is referred to as an analysis waveform because it encapsulates an 
analysis of the original recorded note. Waveform 200 has an attack phase duration 
Dal and a (decay phase d^ation of Da2; r The attack phase includes a set of timing 
marks Tl named Tl[i], the number of time-marks in Tl is named Nal. The decay 
phase includes a set of timing marks T2 named T2[i], the number of time-marks in 
Tl is named Na2. 

[11] : ; Figure 2B is an illustration of a synthesis waveform envelope 202 of a 
synthesized music note having a different , pitch and a shorter duration than 
analysis waveform 200 of Figure 2A. Note, only the upper half of the waveform is 
shown, for simplicity. Synthesized waveform 202 has a total duration Ds. 
According to an aspect of the invention, an attack phase duration Dsl is 
approximately equal in length to Dal. A decay phase is then formed having a 
duration Ds2 = Ds-Dsl. It has now been determined by the present inventor that 
the ear is very sensitive to the attack phase, in which a transition from silence to 
the initial note is made. Therefore, maintaining instrumentally correct aspects of 
the attack phase while varying the pitch is critical for realistic synthesis. 
[12] When the duration of the synthesized tone is shorter than the analysis 

tone, there will be relation processing referred to as type-A on analysis time-marks 
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indexes up to the one corresponding to a sample position equal to the last sample 
position of the synthesis. 

[13] Figure 2C is an illustration of a synthesis waveform envelope 204 of a 

synthesized music note having a different pitch and a longer duration than analysis 
waveform 200 of Figure 2 A. Synthesized waveform 204 has a total duration Ds. 
Again, according to an aspect of the invention; an attack phase duration Dsl is 
formed to be approximately equal in length to Dal even though the pitch is 
different. A decay phase is then formed having a duration Ds2 = Ds-Dsl. 
[14] When the duration of the synthesized tone is ^reiat^r than the analysis 

tone, then there will be relation processing type-A on the Nal time-taarks Tl [i] and 
a relation processing tyjpe-B on the Na2 time^arks T2[i]. The duration of (Ds — 
Dsl) is named Ds2 and corresponds to the fend of the synthesis part of the 
waveform. l , • ; * : r ■ v * 

[15] For type-A, the computation consists of a pitch f modification of the 

analysis waveform. For type^B, the computation consists of a pitch modification and 
a duration extension to be applied only on the T2 time-marks in the decay portion of 
the analysis waveform. This is referred to as "time warpinjg" because the decay 
portion of the analysis waveform is stretched out to match the duration of the 
synthesized waveform. : f * * 

[16] Figure 2D illustrates an alternative method that can be used when the 

duration of the synthesized tone is shorter than the analysis tone, as was discussed 
with reference to Figure 2B. In this case, type B processing is used for the decay 
portion in order to time warp the decay portion and synthesizes a gradual decay 
rather than an abrupt end as was illustrated in Figure 2B. 

[17] Figure 3 is an illustration of a higher pitched note that is synthesized 

from a lower pitched analysis waveform, illustrating repliciation of an analysis 
period, according to an aspect of the present invention. Analysis waveform 300 has 
timing marks 304a-n computed that coincide with each period, such as period 300a. 
The timing marks are used as indexes during the synthesis calculations, which will 
be described in more detail later. Synthesis waveform 302 has a set of timing 
marks 306a-n computed to correspond to the periods that are to be synthesized, 
such as 302a. For each period, samples are selected from the closest time-line wise 
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corresponding analysis period. For example, for synthesis period 302a, samples are 
selected from corresponding analysis period 300a. 

[18] : Since the pitch of synthesis waveform 302 is higher than analysis 

waveform 300, a time skew develops. In order to compensate for this time skew, 
two synthesis periods 312, 314 are formed by selecting samples from the same 
analysis period 310 whenever the time skew becomes approximately one period in 
length. 

[19] In a similar manner , if the pitch of a synthesis waveform is lower than 

analysis waveform 300, a time skew also develops. In this case, an analysis period 
is skipped whenever the time skew becomes approximately one period in length. 
[201 The relation between the analysis time-mark index and the synthesis 

time-mark index is a multiplication factor. The analysis time-mark index has a 
value ranging from 0 to Na-1, where Na = total number of analysis time-marks. 
The synthesis time-mark index has a value ranging from 0 to Ns-1, where Ns = 
total number of synthesis time-marks. If Is is the current synthesis time-mark 
index and la is the current analysis time-mark index, the synthesis will is based on 
waveform extraction of the corresponding analysis waveform located on the time- 
marks la = Is ^ Ks, where Ks is a fractioriaL factor and the multiplication must be 
rounded in order to give an integer index value for Ja. 

For Type-A relation processing, Ks is computed as follows: 

: Ks = ,Ts / Ta ; - K , 

Type-B relation processing, Ks is computed as follows: 
Ks = (Ts*Da2) / (Ta*Ds2) 

[21] For example, assume an analysis waveform recorded at an 8000Hz 

sample rate, the pitch of which is 500Hz and the duration is 50ms. The attack 
portion of the waveform is determined to be approximately the first 20ms, therefore 
the Tl time-mark set is computed and corresponds to 20ms of the beginning of the 
waveform. Accordingly, the T2 time-mark set corresponds to the decay portion of 
the waveform, which in this case includes time-marks in the set [20ms... 50ms], 
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The analysis time-marks are spaced such that each period includes sixteen samples, 
since (8000Hz/500Hz)=16. Therefore, the Tl subset is the set of samples {16, 32, 
144, 160}, the T2 subset is the set of samples {176, 192, 384, 400}. 
[22] Now, in order to synthesize a tone having a duration of 40ms and pitch 

of 1000Hz, then the synthesized waveform will have (8000Hz*40ms) = 320samples. 
For this wave-form, there are 40 synthesis time-marks that include the set of 
samples Ts={8,16,24,32,..., 312,320}. Because the synthesis duration is smaller 
than the analysis duration, Type-A processing is applied. The synthetic music 
waveform period Is is extracted from the analysis waveform located at position 
index la where : - : . r - ^ ^ . 

Ks = Ts/Ta here Ta=16 and Ts^=8 

Ks = 0.5. ' ' : - ^ ; ••. .. . ; 

[23] Therefore, for this example, the, relationship between the synthesis 

period and the corresponding analysis period is: 

Ia = Ks*Is 

la = 0.5 *Is ■ : . - :• . 

[24} In a second example, in order to synthesize a tone having a duration of 

80ms and pitch of 1000Hz, then the synthesized waveform .will have * a Ds = 
(8000*0.080) = 640samples. For this waveform, there are 80 synthesis time-marks 
that include the set of samples Ts={8,16;24,32,..., 632, 640}. Because the synthesis 
duration is greater than the analysis duration, Type-A processing is applied on the 
Tal time-marks and Type-B processing is applied in Ta2 time-marks. The synthetic 
music waveform period Is is extracted for the analysis waveform located at position 
index la where : 

Ksl = Ts / Ta here Ta=16 and Ts=8 

la = Ksl*Is 

Ia = 0.5*Is for ia=0..Nal^l 

and 

Ks2= (Ts*Da2) / (Ta*Ds2) here Da2=30ms and Ds2=60ms 
la = Ks2*(Is - Nal/Ksl) 
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la = 0.25*(Is - Nal/Ksl) for Ia=Nal..Na2-l 

[25] Thus, synthesis periods Is {0...19} will be extracted from the analysis 

period Ia=*),...,9 and corresponds to the synthesized samples {0, ... 159}. Synthesis 
periods Is {20... 79} will be extracted from the analysis periods la = 10,. ..,24 and 
corresponds to the synthesized samples {160,..., 639}. 

[26] Figure 4 is ari illustration of another embodiment of the invention 

illustrating the use of two windows to form a synthesized waveform 410. Each 
window is a Hanning window. A Hanning window is a cosinous digital 
manipulation of a sampled signal which forces the beginning and ending samples of 
the time record tozero* amplitude. Other embodiments of the invention may use 
other known types of windows, such as Hamming, triangular, etc. 
[27] Representative windows 420-422 are shown for illustration; however, 

similar windows are applied continuously along the entire length of the synthesis 
waveform. For each time mark index position, a window is determined that is the 
minimum length of both the local . period of analysis and synthesis around the local 
index [m]. ^ ; ~ ;> • 

Window length = ,WL ^ 

= min (Time(Is[m+l])-Time(Is[m-l]), Time(Ia[m+l])/nme(Ia[m-l])) 
i q ^ (withlafm] = round(Ks^[m-l]) 

[28] This window length covers 2 periods: one before Ia[m] and one after 

Ia[m]. Function "time" gives the absolute position of the sample position in the 
wave files (analysis & synthesis) when the input is the synthesis period index. For 
example: 

Time(Is[40]) = 1000 

means that the 1000 th sample of synthesis corresponds to the 40th synthesis start of 
period. . : 

[29] Once the window length is determined, a function is called for 

computing, with embedded pre-computed tables, the Hanning window for the 
extraction of analysis samples. This function takes the window length as input and 
returns an array of data corresponding to the corresponding window length. For 
example, Win(18) returns a raised cosinous window of 18 samples. 
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[30] Due to the possible large values of Ks, a smoothing operation is applied 

that uses an interpolation between two consecutive analysis extracted periods of 
samples before putting them on the* synthesis time-scale. More precisely, the last 
period of analysis indexed from the previous ia index is used to smooth the current 
synthesis period. The two periods of analysis are weighted and summed before 
being put on the synthesis time scale. The weights are computed with the fractional 
part of the computation F= Is * Ks. The two weights applied on the two analysis 

periods are: • ' v ' : - ' i: — v,; ' f >' J " : ' : " v - 

Wl = (1.0-(Is*Ks-((integei-)(Is*Ks») ^ ;r V- 
W2 = (Is * Ks - ((integerX Is^ Ks»i ; ^ - ir p ; M - r 
The computation uses the non-intege* part; of the product Isf Ks and is performed 
using masks and shifts. Ks is represented in Q9.6 format; a 16bit integer is coded 
with the 9 MSB as integer part and the 6 LSB as fractional part. In another 
embodiment; other formats may be used, such as a floating-point representation, for 
example.' ■ ; --.^,.;--v ,r- ■;■ - ■::/- -\y-. '• • - * 

[31] Thus, for a given synthesis sample, such as^ synthesis sample 414, a 

sample 414a extracted with window 420 from analysis periods 402-403 is weighted 
and combined with a weighted sample 414b retracted with window 421 from 
analysis periods 403-404. ^ 

[32] As discussed earlier, due to time skew, the same analysis periods are 

occasionally reused. For example, for synthesis sample 415, a sample 415a 
extracted with window 421 from analysis periods 404-404 is weighted and combined 
with a weighted sample 415b extracted with window 422 from the same analysis 
periods 403-404. 

[33] This weighting feature is designed for the conditions where a small 

portion of an analysis signal is stored and a long synthesis signal is requested. Then 
the Ks value is very small (for example 0.03) and the weighting then corresponds to 
a smoothing factor instead of having long repetitions of the same analysis windows. 
[34] In another embodiment of the invention, interpolation can also be 

performed to compensate for the fact that generally the exact position of the 
synthesis period does not correspond to a sample boundary. The interpolation uses 
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two extracted analysis windows.- The positions of the synthesis periods are spaced 
; from a time mark Ts that is not an integer; for example 

300Hz=> Ts=8000/300=26.67. 
In this example, the fractional part is: 

FRAC(26.27*3n)- Q.333- ; 
If m=50 and the. two weights are wsl=(l-0.333) and ws2=(0.333), the synthesis 
, samples are then computed as follows: . . . 
•• Ppr- 1 (i*.(-m.Xup tp..CtynL)) 4q„{, . 

XI = W[i] * Analysis_signal[Time(Ia[m3) + i] , 
■-y. ^ X2 - W[i] * Analysis^signal[Time.(Ia[m]X+ i + 1] 

Sy,nthesis_signallTime(Is[in]) .+ i] = wsl*Xl + ws2*X2 

[35] ... ■ Advantageously, the total number of operations is only four multiply 

and one addition per . synthesis sample for the interpolation. When the interpolated 
samples are weighted and combined as shown in Figure 4, then the total number of 
operations per final synthesis sample is doubled but still modest: eight multiply and 
two additions. . , 

[3jSJ ;. . ; In another embodiment, an additional step is performed to improve a . - 

sjmthesized waveform .-that performs a time-reversal operation on selected periods. 
A pseu4o-random number generator is used to decide if the current time-mark 
period is to be swapped. The, first sample of the period to be copied to the synthesis 
/time scale is referred to as A[tm_ia], and tsa is the number of samples extracted 
from analysis. If the current computed period index la is identical to the previous 
computed, one for the last synthesis period due to time skew as described above, 
then time-reversal is considered. If the random number generator gives an even 
value the samples are copied with the respect of the time sequence, that is, the first 
sample is A[tm_ia] and the last one is A[tm_ia+tsa-l]. Otherwise, if the random 
data is odd the time sequence is inverted, such that the first synthesis data is 
A[tm_ia+tsa-l] and the last one is A[tm_ia]. 

[37] Figure 5 is a flow chart summarizing the steps for synthesizing a note, 

according to aspects of the present invention, as described above. In step 500, 
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recordings of the various instruments are made, they are each analyzed, and a set 
of analysis time marks is computed. The annotated analysis waveforms are then 
stored. 

[38] In step 502, a selected note or a melody is received- typically in the 

form of a melody file, which is to be synthesized. A file format for this step will be 
described in more detail later. For each note, k set of synthesis time marks is 
calculated. The following steps are performed for each note. If more than one note 
is to be played in parallel, then the following steps ate performed for each note 
within a time frame to allow parallel play! : \ . ^ * < 

[391 In step 504, for each note an annotated ^halysis waveform is accessed 

as defined by the melody fite^-A ^ time 
marks and the set of synthesis time marks is then computed according to the 
duration of each. If Ds > Da, then type A processing will be used on the attack 
portion and type B processing will be used on the decay portion. If Ds =< Da, then 
type A processing will by used on the Entire synthesis waveforms Coefficient Ks is 
calculated for type A processing, while coefficients Ksl arid Ks2 are calculated for 
type B processing. ■* : '« ' !j '^ 7 "'< "■ ' ;V - ^-^.v:\^ .-s/v' v- \; * 

[401 Step 510 is part of an iteration loop that incrementally computes each 

period of the synthesized waveform. This loop is' traversed for each peribd of the 
synthesized waveform using an index m that is initialized to zero.- During each 
iteration of this step, a set of synthesis samples is computed for the synthesis period 
Is [m-1] . Previous synthesis period Is [m-1] is computed from analysis period la 
= round(Ks*[m-l]) using the cosinous Hanning window described previously. As 
described previously, if the duration of the synthesis waveform is less than or equal 
to the duration of the analysis waveform, then type A processing is used on all of 
the synthesis periods. However ^ if Ds > Da; then type A processing is used for 
synthesis periods within the attack portion and type B' processing is used for 
synthesis periods within the decay portion; 

[41] Likewise, during each iteration of step 512, a set of synthesis samples 

is computed for the synthesis period Is [ml. Type A processing and type B 
processing is performed in accordance with the relative durations of the synthesis 
and analysis waveforms. 
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[42] In an embodiment that includes to compensate for the fact that 

generally the exact position of the synthesis period does not correspond to a sample 
boundary, as described above, an interpolation calculation is included in step 510 to 
compute synthesis period Is[m-1] and in step 512 to compute synthesis period Is[m]. 
[43] Step 520 determines if time reversal should be considered for this 

iteration. If, in step 512, round(Ks*m) = round(Ks*[m-l]), then a random reversal 
of the synthesized samples within the current period Is[m] is invoked. The random 
reversal is based; on- a pseudo random mimber generator that is tested in step 522. 
If the random- niixiiber is odd, then time reverse the Is [m] set of samples, otherwise 
do not perform a ? time reverse. . - ~u ■* {, ± 
* • [44] * In step 524; if no time reversal is to be done, then each sample of 

previous the synthesis period Is[m-1] is weighted by weighting factor Wl, where 
Wl=(1.0-([ml 4: Ks-((mt)([m]*Ks))). Each sample of the current synthesis period 
Is[m] is weighted by weighting factor ^, where W2=([m]^Ks-((int)([m]*Ks))). The 
i. . i" .' results are added together sample- wise to form a final version of current synthesis 
period Is [m] and then added to the time scale. 
For example: if Ks = 0.3, m=454, 
then^l=(1.0^((454*0- 3):.:int(454*0.3)), . .* : 

W2=(454*0.3)-int(454*0.3) 
. , ■;■ =0.2 - >, - .'- - - - 

[45] - If a time reversal is to be done, then step 526 is performed instead of 

524. Weighting is performed the same as for step 524; however, the set of samples 
for the current synthesis period from step 512 are time reversed prior to combining 
with the samples from the previous synthesis period from step 510. 
[46] - Step 530 is the end of the iterative loop. Index m for Is is 
incremented by one and the loop beginning with step 510 is repeated until the final 
synthesis period of the note is reached. The sample set Is[m] that was calculated in 
step 512 is saved and is used as the "previous synthesis period" for the next pass 
through the loop so that no additional calculations need be performed in step 510. 
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[47] Figure 6A is a flow diagram ^ illustrating how a set of analysis 

waveforms are collected. In step 600, a single note from an instrument is sampled 
to form an instrumentally correct digital analysis waveform. The sampling rate is 
selected according to the expected use. . For telephone type devices, a sampling rate 
of 8kHz is typically used. For a high quality audio synthesizer, a sample rate of 
40khz might be used, for example. ; \ > 

[48] In step 602, the sampled digital waveform is analyzed to determine the 

duration of an attack portion and the duration of sl decay portioh> In the present 
embodiment, this characterization . is -*J performed - by, displaying the • sampled 
waveform on video display device and visually selecting a: time point at which the 
attack portion is complete.^ Another embodiment may automate this step using a 
waveform analysis filter, for example. t ^ ; > ^ ^ ^ ^ n 
[49] A set of timing , marks is also . calculated during step 602 that 

corresponds to the period boundaries of the analysis waveform. For a non-periodic 
waveform, timing marks can be assigned at regular intervals. A set of timing 
marks Tl is computed for the attack ;portion and a set : ., of timing marks T2 is -j 
computed for the decay portion. - ; ^ a< c 

[50] The digital waveform arid the. duration information and the two sets of 

timing marks are then stored in a file as an annotated analysis waveform for later 
use. - : > . : 

151] Figure 6B is an illustration of an orchestra file that includes a set of 

analysis waveforms for several different instruments. Each entry in the orchestra 
file is an annotated analysis waveform, as described above, that includes a digitized 
analysis waveform, duration information and timing marks. The synthesis method 
described herein can produce good quality synthesized notes over a range of three to 
five octaves for some types of instruments. Typically, a wide range of instruments 
can be synthesized using these techniques over a range of approximately +/- one 
octave from the analysis waveform. Therefore, for a telephone type device that has 
a bandwidth of approximately 4 kHz, only two analysis samples are required for 
each instrument, at 500 Hz and at 2000 Hz. For instrument types that may not 
normally produce a broad range of notes, only a single sample may suffice, such as 
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for a bass 620 that does not produce higher notes or for a flute 622 that does not 
produce lower notes. , 

[52] : Advantageously, a wide range of instruments can be represented in an 

orchestra file in a relatively small amount of memory ; 

[53] Figure. 7 is a block diagram of a digital system that includes an 

embodiment of the present invention in a megacell core 100 having multiple 
processor cores. In the interest of clarity, Figure 1 only shows those portions of 
megacell 100 that are relevant to an understanding of an embodiment of the 
present invention. Details of general construction for DSPs are well known, and 
may be found readily, elsewhere: For example, U.S. Patent 5,072,418 issued to 
- Frederick Boutaud^et al, describes a DSP in detail. U.S f Patent 5,329,471 issued to 
OGary Swoboda; et al. describes in detail how to test and emulate a DSP. Details of 
I portions of megacell 100 relevant to an embodiment of the present invention are 
explained^ in sufficient detail herein below, so r as to enable one of ordinary skill in 
;;. the microprocessor art to make and use the invention. 

[54] c ; , . : ; - Referring; again to Figure 7* megacell 100 includes a control processor 
; (MPU) 102 with a 32-bit core 103 and a digital signal processor (DSP) 104 with a 
^SP core 1G5 that share a block of memory 113 and a cache 114, that are referred 
t , to. as a level two (L2) memory subsystem 112. \ A traffic control block 110 receives 
^ transfer requests from a host processor connected to host interface 120b, requests 
from control processor 102, and transfer requests from a memory access node in 
DSP 104. The traffic control block interleaves these requests and presents them to 
the shared memory and cache. Shared peripherals 116 are also accessed via the 
traffic control block. A direct memory access controller 106 can transfer data 
between an external source such as off-chip memory 132 or on-chip memory 134 and 
the shared memory. Various application specific processors or hardware 
accelerators 108 can also be included within the megacell as required for various 
applications and interact with the DSP and MPU via the traffic control block. 
[55] External to the megacell, a level three (L3) control block 130 is 

connected to receive memory requests from internal traffic control block 110 in 
response to explicit requests from the DSP or MPU, or from misses in shared cache 
114. Off chip external memory 132 and/or on-chip memory 134 is connected to 
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system traffic controller 130; these are referred to as L3 memory subsystems. A 
frame buffer 136 and a display device 138 are connected to the . system traffic 
controller to receive data for displaying graphical images. A host processor 120a 
interacts with the external resources through system traffic controller 130. A host 
interface connected to traffic controller 130 allows access by host 120a to external 
memories and other devices connected to traffic controller 130. Thus, a host 
processor can be connected at leVel threfe or at level two ins various embodiments. A 
set of private peripherals 140 are connected to the DSP^ while another set of private 
peripherals 142 are connected to the MPU; ■ ^ , ' ^ 

[56] Although the invention fiiids particular- application to Digital Signal 

Processors (DSPs); implemented, fot exarriple* m an Application Specific Integrated 
Circuit (ASIC), it also finds application to other forms of processors: An ASIC may 
contain one or more megaceUs which each^ include custom -designed functional 
circuits combined with pre-designed functional circuits provided by a design library. 
[57] Figure 8 is a flow chart illustrating synthesis of a melody on the digital 

system of Figure 7 according to an aspect of the present* invention. : Software 
executing on MPU 102 responds to a user request or other stimuli to select a melody 
for synthesis. In step 800, MPU 102 loads thearialysis waveforms and analysis time 
marks into shared memory 112. If the request is for jtist at single instrument, then 
only the annotated analysis waveforms for the selected instrument are loaded. For 
a more complex melody, an Entire orchestra file is loaded; The orchestra file is 
maintained in the L3 memory subsystem. ' 

[58] In step 802, MPU 102 loads a file that contains a requested musical 

score into shared memory 112. A musical score file is referred to herein as an E2 
file; - - •• • 

[59] The E2 file format is a compressed binary file in order to use as least 

possible memory in the MPU address space. The data rate is about 4 bytes per 
synthesized note. This size can be greater with optional sound generation effects 
like: pitch bend, volume tremolo and vibrato. 

[60] The E2 file format, for each note there is an 8-bit data byte indicating 

two things: the first seven bits is a time stamp indicating the time interval in 20ms 
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periods before loading the current note event; and the eighth bit is an indicator of 
an extended format for the following data;; 

[61] The time stamp byte ; is followed by two bytes (16 bits) of note 

definition data haying the. following format: six bits for frequency selection, three 
bits for amplitude, three bits for the analysis wave selection, and four bits for the 
duration. 

[621 If the extended format bit is set then these two bytes are followed by 

four additional bytes used for sound effects control. 

u [63] : The MCII reads the first byte of the data stream, then waits a time 

period according to the time stamp before loading the dual port memory interface 

; ;v. with the note definition data: two bytes or six bytes if the extension bit is set. Then 
the MCU reads the next time stamp byte indicating a delay for the next note before 
loading the next set of note definition data. For notes to be played in parallel, the 
. time delay could be zero. 
[64] Jn step 804, DSP 104 reads each set of note definition data provided by 

• r the MPU from the E2 file and computes the frequency, amplitude, and duration of 

; each note to synthesize using the respective fields in the two byte note definition 

; , data. DSP 104 then computes a set of synthesis time marks for each note. 

[65] In step 806, the DSP computes the relation between the analysis and 

synthesis time marks, as described previously, by selecting an analysis waveform of 
: an* instrument type specified by the three bit wave selection field in the note 
definition data^ Where there is more than one analysis waveform for the specified 
instrument, selection is further based on selecting an analysis waveform whose 
frequency is closest to the frequency specified for the synthesized note. 
: [66] In step 808, the DSP computes the synthesis samples for the requested 

note and applies sample weighting and sample time reversal to improve the quality 
of the synthesized note, as described previously with reference to Figure 5. The 
synthesized samples are then written to an audio conversion interface for playing. 
The audio conversion interface is included in the set of peripherals 140 that are 
connected to the DSP. 
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[67] In step 810, a check is made to see if the last note definition data has 

been received from the MPU. If another note request is pending, the loop is 
repeated using the new note definition data. 

[68] Advantageously, since the synthesized notes' are played iri real time as 

they are generated, only a vanishingly small buffer area is required to support the 
synthesis operation. 

Digital System Embodiment r I:* > w >.i ^ ' ; 

[691 Figure 9 illustrates an exemplary implementktion of the invention in a 

mobile telecommunications device, such as a mobile' personal digital assistant 
(PDA) 10 with display 14 arid integrated input sensors 12a, 12b located in the 
periphery of display 14. As shown in Figure 9< digital system 10 includes a 
megacell 100 according to Figure 1 that is connected to the input sensors 12a,b via 
an adapter (not shown), as an MPU private peripheral 142; A stylus or finger can be 
used to input information to the PDA via input sensors 12a,b. Display 14 is 
connected to megacell 100 via Ideal frame buffer similar to frame buffer 136. 
Display 14 provides graphical and video output iri overlapping windows, such as 
MPEG video window 14a y shared text document window 14b arid three dimensional 
game window 14c, for example. ■ * : ^ , : ■ ; - - ; m — . - - . . ^ • 

[70J Radio frequency (RF) circuitry (not shown) is connected to an aerial 18 

and is driven by megacell 100 as a DSP private peripheral 140 and provides a 
wireless network link. Gonnector 20 is connected to a cable adaptor-mndem (not 
shown) and thence to megacell 100 as a DSP private peripheral 140 provides a 
wired network link for use during stationary usage iri an office environment, for 
example. A short distance wireless link 23 is also "connected" to earpiece 22 and is 
driven by a low power trarismitter (not shown) connected to megacell 100 as a DSP 
private peripheral 140. Microphone 24 is similarly connected to megacell 100 such 
that two-way audio information can be exchanged with other users on the wireless 
or wired network using microphone 24 and wireless earpiece 22. 

[71] Megacell 100 provides all encoding and decoding for audio and 

video/graphical information being sent and received via the wireless network link 
and/or the wire-based network link. 
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[72] A^synthesized -melody that is written by the DSP to an audio 

conversion interface can be listened to via wireless earpiece - 22. Similarly, a 
speaker or a set of speakers can be connected to the audio conversion interface and 
thereby play the synthesized melody. 

[73] It is contemplated, of course, that many other types of communications 

systems and computer systems may also benefit from the present invention, 
particularly those relying on battery power. Examples of such other computer 
systems include portable computers, smart phones, web phones, and the like. As 
power dissipation and processing performance is also of concern in desktop and line- 
powered computer systems and micro-controller applications, particularly from a 
reliability standpoint, it is also contemplated that the present invention may also 
provide benefits to such line-powered systems. 

[74] This music synthesis technique can be applied to many different kinds 

of applications. For example, for various types of electronic musical instruments, 
one analysis wave is recorded for each musical octave scale. Advantageously, the 
algorithm plays all the twelve half-tones of the scale. 

[75] Another embodiment can be used in electronic games to play the music 

used in games. Advantageously, memory requirements and processor resources are 
minimized by the algorithm described herein. 

[76] In another embodiment, cellular and fixed-line phone will use this 

technique in for playing pre-selected or customized ringing melodies. 

[77] As used herein, the terms "applied," "connected," and "connection" 

mean electrically connected, including where additional elements may be in the 
electrical connection path. "Associated" means a controlling relationship, such as a 
memory resource that is controlled by an associated port. The terms assert, 
assertion, de-assert, de-assertion, negate and negation are used to avoid confusion 
when dealing with a mixture of active high and active low signals. Assert and 
assertion are used to indicate that a signal is rendered active, or logically true. De- 
assert, de-assertion, negate, and negation are used to indicate that a signal is 
rendered inactive, or logically false. 
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[78] While the invention has been described with reference to illustrative 

embodiments, this description is not intended to be construed iii a limiting sense. 
Various other embodiments of the; invention will be apparent to persons skilled in 
the art upon reference to this description, * 




******* 
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What is Claimed is: 

1. A method of synthesizing music in a digital system comprising: 
accessing a digital analysis waveform having a first duration, a first pitch, a 

first attack portion and a first decay portion; 

determining a second duration and a second pitch for a synthesis waveform; 

computing first timing marks for the analysis waveform such that the first 
timing marks correspond to periodicity of the analysis waveform; 

computing second timing marks for the synthesis waveform such that the 
second timing marks correspond to periodicity of the synthesis waveform; and 

calculating samples for each period of the synthesis waveform defined by 
adjacent second timing marks using samples selected from a corresponding period 
of the analysis waveform defined by adjacent first timing marks to form the 
synthesis waveform having the second pitch, the second duration, a second attack 
portion and a second decay portion. 

2. The method of Claim 1, wherein the step of calculating samples for 
each period further comprises: 

calculating a set of samples for a period m using a first cosinous window; 
^ - calculating a set of samples for a period m-1 using a second cosinous window; 
aind 

combining the set of samples for period m and the set of samples for period 
m-1 using a weighting function. 

3. The method of Claim 2, wherein the first cosinous window operates on 
two adjacent periods and the second cosinous window operates on two adjacent 
periods shifted by one period from the first cosinous window. 

4. The method according to Claim 2 or Claim 3, further comprising the 
step of: 

reversing a selected one of the set of samples before the step of combining the 
sets of samples. 
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5. The method according to Claim 4, wherein the step of reversing is 
performed only when two consecutive periods of the synthesis waveform are formed % 
using same periods of the analysis waveform; and 

wherein the step of reversing is responsive to a random number generator. 

6. The method according to any preceding Claim, wherein the step of 
calculating samples comprises forming the synthesis waveform such that the second 
attack portion has a duration approximately, equal to a duration of the first attack 
portion. 

- / 7;, Trie method according to any preceding Claim, wherein the step of 
calculating samples comprises forming the synthesis waveform such that the second 
decay portion is form ed by time warping the first decay portion. 

8. The method according to any preceding Claim, wherein the second 
pitch is selected from a range of at least plus or minus one octave around the first 
pitch. 

/ 

/ 

9. The method according to any preceding Claim, wherein the step of \ 
accessing an analysis waveform comprises selecting from a plurality of 
instrumentally correct digital waveforms corresponding to a plurality of 

instruments. I 

i 

10. The method according to Claim 9, wherein for at least one of plurality 
of instruments, the instrumentally correct digital waveforms include not more than 
one waveform for a range of at least two octaves. 

11. A digital system, comprising: 

a memory for holding a plurality of instrumentally correct digital waveforms 
corresponding to a plurality of instruments; 
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a first processor connected to the memory, the first processor operable to 
store a musical score in the memory; and 

a second processor connected to the memory, the second processor operable to 
synthesize a melody signal in response to the musical score using the method 
according to any preceding claim for each note of the melody, 
and 

12. The digital system of 

an audio device connected tbi the 'second processor for playing the 
synthesized melody signal. 

13. The digital system according to Claims 11 or Clirim 12 wherein the 
digital system comprises a personal digital assistant, further comprising: • / 

a display connected to the second processor via a display adapter; 

radio frequency (RF) circuitry connected to the CPU; and 

an aerial connected to the RF circuitry. ' 
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Programmable Melody Generator 

Abstract 

A digital system and method of operation is provided in which musical notes 
and melodies are synthesized. The operation done for music synthesis is based on 
time domain processing of prerecorded waveforms, referred to as analysis 
waveforms. The computations are done using time-marks, which is a set of digital 
sample positions of the analysis waveform indicating the starting position of each 
period of the fundamental frequency or an arbitrary position for non-periodic 
analysis waveforms. The algorithm defines on a time scale the time-marks of the 
synthesis waveform. The synthesis is based on making a relation between the 
analysis time-marks and the synthesis time-marks. The synthesis waveforms are 
built with the extraction of small portions of signal located at corresponding time- 
mark positions of the analysis waveform and adding them to the corresponding 
synthesis time-marks on the synthesis time-scale. This extraction is done with the 
multiplication of a windowing pattern, such as a cosinous Hanning window, to the 
analysis samples. 
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